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Abstract 

Mobile technology has been increasingly adopted in science education. We generally 
assume that more innovative use of mobile technology leads to a greater learning 
outcome. Yet, there is a lack of empirical research to support this assumption. To fill in 
the gap, we drew upon data from 803 high school students who had used mobile devices 
for five months in physics classrooms. By using the SAMR model (ie, Substitution, 
Augmentation, Modification, and Redefinition), we distinguished their uses into two 
levels: Substitution (replacing traditional instructional approach with mobile technology 
without functional improvement) and augmentation (enhancing instruction with 
affordances provided by mobile technology). Using Hierarchical Linear Modeling 
analysis, we found that the augmentation-level of use was positively correlated with the 
physics learning outcome, but the substitution-level of use was not. We further identified 
four sub-types of uses within the augmentation level. We found that after-school 
remediating activities and student-teacher displaying activities were positively correlated 
with student physics achievement, but teacher-assigned activities had no significant 
correlation and learning aid activities had a negative correlation with the learning 
outcome. This study provided empirical evidence to support the assumption that a 
higher level of mobile technology use may be related to a greater learning outcome and 
that the impact of mobile technology may be determined by multiple factors such as 
who initiates the use and whether the use enhances or distracts students' knowledge 
construction. 


Introduction 

Mobile technology offers great affordances to support science learning (Crompton, Burke, 
Gregory, & Grabe, 2016; Zydney & Warner, 2016). With the capability to store a vast amount of 
learning content (eg, eBooks, audios, and videos) on a single device, mobile technology makes it 
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Practitioner Notes 
What is already known about this topic 


* Multi-functional mobile technology has been increasingly adopted in science learning, 
but there is a lack of research on what kinds of functions or usages tend to be more 
effective. 

* Prior research suggested that we should focus on better understanding how technology 
is integrated into classrooms, rather than pursuing the latest technology for its own 
sake. 

* Prior research revealed that mobile technology could be used to support instruction at 
different levels, but it is unknown about the relationship between the levels of mobile 
technology use and student learning outcomes. 


What this paper adds 


* This study is among the first that applied the SAMR model to understand the levels of 
mobile technology use in high school physics classrooms and its impact on learning, 
drawing upon relatively large sample size. 

* This study found that mobile technology use at the augmentation level is associated with 
greater student learning outcomes, but the use at the substitution level is not associated 
with learning outcomes. 

* This study found that the impact of mobile technology use was further mediated by mul- 
tiple pedagogical factors, such as who initiated the use. Student-initiated and knowledge 
construction oriented mobile technology use tend to be more effective. 


Implications for practice and/or policy 


* This study suggests that we should encourage higher levels of mobile technology use in 
classrooms to support learning. Substituting traditional instructional methods with mo- 
bile technology might increase the user-friendly experience but not necessarily improve 
learning. 

* [n particular, educators should encourage student-initiated mobile technology use for 
learning and design activities that use mobile technology for knowledge construction. 


easier for students to access information and manage educational resources. A growing number 
of science education apps allow students to conduct science experiments, record data, and ma- 
nipulate science simulations (Zydney & Warner, 2016). Internet-connected mobile devices also 
enable anytime and anywhere teacher-student and student-student interaction and communi- 
cation. Teachers can use mobile devices to provide timely feedback to students, and students can 
use their mobile devices for self-directed remedial learning (Zhai, Zhang, & Li, 2018). 


Nonetheless, mobile technology research in science education has found inconsistent effect sizes 
on student achievement. A recent meta-analysis by Sung, Chang, and Liu (2016) that exam- 
ined 27 mobile technology studies in science learning reported effect sizes ranging from .392 to 
.738. However, Zhai, Sun, and Guo (2016) found a negative effect. Sung and colleagues (2016) 
suggested that the variation of these effect sizes may be due to how mobile technology is used in 
science classrooms. For example, sometimes mobile devices are simply used to replace the tra- 
ditional instructional methods without adding functional improvement (eg, reading an article 
on paper versus reading the same article on screen). Other types of use aim to enhance science 
instruction by taking advantage of the affordances of mobile technology (eg, using interactive 
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science simulations; Romrell, Kidder, & Wood, 2014). This difference might be one of the reasons 
for the varied effects, but no data verified the assumption. 


Researchers call for innovative technology-integrated learning to “make it possible to adopt new 
and arguably better approaches to instruction and/or change the content or context of learning.” 
(Lawless & Pellegrino, 2007, p. 581). Itis generally assumed that a higher level of technology use 
in science learning is associated with a greater student learning outcome (Zhai, Xiang, & Mu, 
2015). However, to our knowledge, there is no empirical evidence to support this assumption, 
especially for mobile technology (Sung et al., 2016). The lack of evidence is primarily due to the 
demanding requirements of conducting comparison studies. For example, researchers have to 
control many major covariates such as participants, devices, software, and content, as well as 
classroom settings. It is often difficult to meet these requirements to isolate the impact of mobile 
devices such as smartphones or tablets without comparable research sites. 


To address this issue, this study employed Hierarchical Linear Modeling (HLM) to control for con- 
founding variables due to classroom setting differences. Participants in this study used the same 
mobile device (ie, tablet) with multi-functional educational apps to support different learning 
activities. We first adopted the SAMR model, which divided the extent of technology integrating 
learning into four successive levels: Substitution, Augmentation, Modification, and Redefinition 
(Puentedura, 2006). Then, we examined the relationship between the frequency of using mobile 
technology with varying levels of integration according to the SAMR model and its effects on 
high school physics learning. We further identified four types of uses with specific pedagogical 
characteristics in the Augmentation level and examined whether and how these four types of 
uses predicted students’ physics achievement. This study explored two research questions: 


* Does a higher level of mobile technology use in high school physics classrooms predict greater 
achievements? 

* How does the effect size for physics achievement differ across the four specific types of peda- 
gogical uses within the augmentation level? 


A model indicating the levels of technology-integrated learning 

The SAMR model 

Despite the great affordances that technology can potentially offer to support learning, technol- 
ogy is often used as a simple replacement of existing practice (Harris, Mishra, & Koehler, 2009). 
To account for the range of how technology is used in classrooms, Puentedura (2006) developed 
a four-level model for technology-integrated learning that includes Substitution, Augmentation, 
Modification, and Redefinition. 


At the substitution level, technology is used to substitute conventional teaching or learning prac- 
tices directly without functional improvement. For example, Lan and colleagues (2012) stud- 
ied the use of mobile technology for asynchronous online discussion. The mobile devices in this 
study merely replaced desktop computers and added no functional improvement. Even so, Lan 
et al. observed an increased frequency of use and active student participation in the discussion 
boards. At the augmentation level, mobile technology is used to support learning with functional 
improvement. For example, Pfeiffer and colleagues (2009) conducted a quasi-experimental study 
using mobile devices to display video, audio, and static screenshots of a field guide to help stu- 
dents identify species of fish during a snorkeling field trip. The mobile devices not only replaced 
the printed field guide but also provided functional improvement (eg, enriched multimedia mate- 
rials to stimulate students' imagination and engage them in learning activities). The participants 
in this study showed higher knowledge gain than did those who used the traditional field guide. 
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The next level, modification, involves using technology to redesign conventional instructional 
practice, resulting in significant changes in functional development. For example, in a study on a 
redesigned geomorphology course by Cornelius (2011), students used mobile devices to receive 
course information and made decisions in a real-life simulation of a flood disaster. Students 
reported enjoying the learning activities on how to prevent a flood. In this study, mobile tech- 
nology was used to engage students in interactive real-life simulations, a capability not available 
in textbooks, to support student learning. Redefinition, the highest level of integration, indicates 
that the curriculum is redesigned around innovative mobile technology affordances that are oth- 
erwise not possible in a traditional learning environment, such as creating a media-rich digital 
story in a language arts class. This level of integration has rarely been implemented in science 
learning (Romrell et al., 2014). 


Atthe substitution and augmentation levels, technology functions as an enhancement to the tra- 
ditional instructional approach, whereas at the levels of modification and redefinition, technol- 
ogy transforms instruction. This SAMR model has been adopted by researchers to characterize 
the levels of integration in mobile technology studies (Romrell et al., 2014; Zhai, et al., 2018), but 
there is a lack of research on how different levels of integration are associated with the learning 
outcome. 


Pedagogy-mediated technological integration 

The SAMR model entails an implicit assumption that a higher level of integration will lead to 
greater student achievement. However, Hamilton and colleagues (2016) argued that the actual 
implementation of technology in classrooms is more complicated than what the SAMR model 
could capture. The SAMR model describes how technology can be used at different levels in 
general but does not tell us precisely in what ways mobile technology should be used in class- 
rooms to receive maximum benefits. Poorly implemented lessons are unlikely to produce greater 
achievement even with the intention of a higher level of technological integration (Hennessey, 
Ruthven, & Brindley, 2005; Windschitl & Sahl, 2002). 


To understand the relationship between the level of technological integration and students’ 
learning achievement, this study paid attention to the pedagogical characteristics of the spe- 
cific uses within each integration level. We consider the pedagogical approach a broad concept 
that mediates the levels of integration. Therefore, it is possible to find similar or identical peda- 
gogical approaches in different levels of integration (Bray & Tangney, 2017). The pedagogical 
approaches in science education with technology include some well-established ones such as 
individual constructive, didactic, or collaborative learning determined by the roles of students 
and teachers in a learning context (Hwang, Shi, & Chu, 2011; Lindsay, 2016). Research shows 
that students could “self-direct their learning and access content relevant to them” (Friedel, Bos, 
Lee, & Smith, 2013, p. 3716) and consider learning as an actively constructing process. Students 
under such a pedagogical approach might seek help with technology as needed. In contrast, 
under the didactic approach students’ personal needs and initiatives are often dismissed, while 
“students read or wait to be told exactly how they are to perform” (Lord & Orkwiszewski, 2006, 
p. 342). Collaborative learning involves interactions among peers in groups or between teachers 
and students in a teacher-directed instructional approach in which teachers play a role to facili- 
tate students’ communication, which usually requires multiple rounds of information exchange. 
Technology facilitates this information exchange process. The extent, intention and ways of stu- 
dent engagement under this pedagogical approach are considered crucial to their learning out- 
come (Resta & Laferriére, 2007). 


Because many factors might impact the specific pedagogical approach regarding using tech- 
nology, the current study did not aim to capture all the features of the pedagogical approaches 
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described above. Instead, we focused on four critical aspects: the pedagogical purpose, student 
activity, teacher activity and learning context for the specific uses. These aspects were highlighted 
in many mobile learning studies to provide a comprehensive picture of the pedagogical features of 
the classroom learning (Lindsay, 2016; Rambe & Bere, 2013). 


Integration of mobile technology in physics learning 

Although mobile technology has been increasingly adopted in science education, only limited 
studies focused on physics learning. According to the latest review on mobile learning in science 
education (Zydney & Warner, 2016), there were only four such studies. Another review study on 
mobile science learning reported only three studies (Crompton, Burke, Gregory, & Grábe, 2016). 
We conducted a broader review of mobile technology and found ten studies, of which there were 
mainly four types of uses in physics learning—sensor-based measurement (eg, Oprea, & Miron, 
2014), assisting classroom learning activities (eg, Liu, Wu, Wong, Lien, & Chao, 2016), formu- 
lating ubiquitous learning (eg, Hwang, Yang, Tsai, & Yang, 2009), and using multi-functional 
systems (Zhai, et al., 2018). Among the ten studies, two used mobile technology at the substitu- 
tion level, six at the augmentation level, and two at both levels. The primary pedagogy in these 
studies was individually construction approach—self-directed learning (4 out of 7) and collab- 
orative approach (4 out of 7). All found positive effects except for three studies that did not have 
empirical data (see supplementary material Table S1). 


Methods 

Materials and procedure 

This study is an extension of our prior project (Zhai, et al., 2018) in which we developed an in- 
teractive learning management system (ILMS) and installed compatible multi-functional apps 
on tablets for students and physics teachers in a high school in China. The ILMS, used in the 
classrooms with student one-to-one tablets, teacher tablets, an interactive whiteboard, and an 
educational cloud platform, supports students and teachers with functions in classroom con- 
trol, resource management and distribution, response collection, screen display, instructional 
preparation, homework assignment and collection, and online assistance. In our prior study, 
we examined the extent to which mobile technology was used in physics learning by identifying 
twenty most frequent uses of mobile devices at two integration levels (Puentedura, 2006). The 
lower integration level, which includes three specific uses (Picture uploading, Discussion aid, 
and My textbook), was categorized as substitution, while the higher integration level (ie, augmen- 
tation) includes 17 uses (eg, Screen notes; see Table 1). 


This study first examined the correlation between the frequency of use with varying levels of 
mobile technology use and students' physics achievement. We then explored four types of uses 
with different pedagogical characteristics in the Augmentation level. We further examined the 
consistency of the above correlation across the four types of uses. 


Participants 

We recruited 831 10" grade students (428 males and 403 females) from 28 classrooms in a high 
school in China. These students opted for the study. The average age ofthe students was 15 years, 
ranging from 14 to 16 years. We collected data from these students in their first semester in high 
school. Prior to this study, none of the students had used the ILMS. They were trained by their 
teachers to use the mobile devices after entering high school and frequently used this system 
both in class and after school for physics learning (Zhai et al., 2018). After cleaning up the invalid 
responses, we got 803 valid samples (ie, 409 males and 394 females) analyzed in the study. 
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Instruments 


Frequency of Mobile Technology Use Questionnaire 


We developed a questionnaire to survey students’ use frequency of the tablets regarding the 
eleven in-class uses and nine after-school uses on a 5-point Likert-scale (Specific procedure of 
the development refers to Zhai et al. (2018)). The eleven in-class items include response options 
of (1) in every class, (2) in most classes, (3) in about half of the classes, (4) in less than half of the 
classes, and (5) seldom. The nine after-school items include response options of (1) every day, (2) 
2-3 times a week, (3) once a week, (4) once a month, and (5) seldom. The Cronbach’s coefficient 
for the 20 items was .95, indicating a robust internal consistency. 


Test for Physics Achievement 


After students entered high school, they took a physics test on physics learning. This score was 
included in our HLM analysis to control for prior student achievement. They took another phys- 
ics test administered by the school district to assess their achievement at the end of the semester. 
This score was considered as the outcome. The two tests were designed to examine students’ 
ability to solve physic problems rather than memorizing of physics knowledge. Both tests have 
identical item formats: ten multiple-choice items, three short-answer essay items, and four open- 
ended items. The score of the items for each test adds up to 100. The Cronbach's coefficient for the 
two tests is 0.79, indicating sufficient reliability. 


Survey for physics interest 


In order to measure student physics learning interest, we employed an adapted science interest 
scale (Lamb et al., 2012). Four items that shared the same 5-point Likert-scale with 1 represent- 
ing strong disagreement and 5 representing strong agreement, were selected and adapted to tap 
student physics learning interest. An example of the item stem includes “Compared to pursuing 
test scores, I prefer the inquiry process of learning physics.” The similar procedure has been done 
by Zhai et al. (2018) and found valid. The Cronbach’s coefficient for this survey is .85, suggesting 
a robust internal consistency. 


Identifying the Pedagogical approaches within the Integration Level 

Concerning no strong theoretical assumption regarding the patterns of using mobile technology, 
we used a random split approach (Fabrigar, Wegener, MacCallum, & Strahan, 1999) to identify 
the pedagogical approaches of technology use. We randomly split the original survey data (from 
the 803 participants) into two segments, and then conducted exploratory factor analysis (EFA) 
to identify the patterns in SPSS. To verify or confirm the patterns identified, we further applied 
confirmatory factor analysis (CEA) in Mplus. With the first segment (402 samples), we conducted 
EFA for the three substitution uses and the 17 augmentation uses respectively. In the prerequisite 
examination of the three substitution uses, the Kaiser-Meyer-Olkin measure of sampling ade- 
quacy is .615, above the commonly recommended value of .6, and Bartlett’s test for sphericity 
is significant (y? (3) = 344.42, p < .05), which indicates the variables are related, thus a factor 
analysis is applicable for the data. The communalities of the three items are all above .40, which 
indicates the proportion of variation in these uses is sufficient to explain the latent factor. Given 
these indicators, it is suitable to conduct EFA for these uses. We employed principal components 
analysis to compute the latent factors of the uses and found, as indicated by the eigenvalues, 
only one factor explained 58.51% of the variance. The primary factor leading to the three uses is 
.83, .80 and .66, respectively. This result indicates consistency across the three specific uses. The 
three uses (ie, Picture uploading, Discussion aid, and My textbook) all belong to the substitution 
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level but have somewhat different pedagogical features. The picture uploading is considered a 
student-teacher collaborative pedagogy because it helps students to share their work with their 
teachers in a timely manner. Discussion aidis also a collaborative pedagogy, but it usually happens 
among peers in the classroom, whereas my textbook is a self-directed use after school. Although 
the three uses have different pedagogical features, the frequency of use indicates that they are 
under a single latent factor. 


For the 17 augmentation uses, we first conducted EFA to explore their pedagogical approaches 
within the 402 samples. The Kaiser-Meyer-Olkin measure of sampling adequacy is .85 and 
Bartlett's test for sphericity is significant (y? (136) = 568.50, p < .05), which indicates the vari- 
ables are correlated. In addition, communality for each specific use is higher than .4 (see Table 1), 
indicating the 17 uses are suitable to conduct EFA. The eigenvalues computed by principal com- 
ponent analysis indicate that there are four latent factors, which account for 69.7696 of the total 
variance. Using Varimax and Oblimin rotations, with four factors, the principal component anal- 
ysis defines the factor structure, as displayed in Table 1, in which each use with a primary loading 
over .40 is specific on only one of the four factors except for three uses. These three uses—preview 
learning guide, review learning guide, and screen note—have factor loadings above .40 on two fac- 
tors, indicating they were not used in a single pattern. Therefore, we eliminated these three uses. 
To confirm these four patterns, we conducted CFA using the other half of the data (401 samples). 
Results indicate that the four-factor model has sufficient model fit,as y? /df —4.10( p <.001), 
RMSEA =. 088 (Probability < .001), CFI = .95, and TLI = .93, which supports the recognized four 
pedagogical approaches. 


To better understand the pedagogical approaches of the augmentation uses, we labeled the four 
patterns involving the remaining 14 uses that were clustered under the four latent factors based 
on their pedagogical characteristics as student-teacher displaying activities, teacher-assigned activi- 
ties, after-school remediating activities, and learning aid activities (see Table 1). The student-teacher 
displaying activities are those activities involving students' displaying or submitting their work 
in response to the request of their teachers. The teachers then respond to students' submission. 
Thus, this use is considered a classroom student-teacher collaborative pedagogy. One such exam- 
ple is Doodling, which requires students to write or draw directly on the tablet screen and to submit 
results to their teacher; teachers then respond to student submissions by selecting a few examples 
and demonstrate them to the whole class. Teacher-assigned activities are those directed by teach- 
ers, who assign or prepare the materials for students (eg, Screen broadcast is directed by teachers) 
either in class or after school. Students often dutifully respond to their teacher's request, but there 
is a lack of interactions as in the student-teacher displaying activities. After-school remediating activ- 
ities are those learning activities that happen after school and are self-directed. For instance, the 
class note is a digital notetaking function used by the students to review, modify, tag, and search 
their notes, and this use is initiated by the students themselves. Learning aid activities are mostly 
used for accumulating learning materials during other learning activities (eg, teacher lecture), 
and are rarely used for knowledge construction. For example, (in-class) building/using incorrect 
item set often happens in the middle of other classroom activities (Table 1). 


Statistical analysis 

To answer our two research questions, this quantitative study used students' post-physics 
achievement as the outcome and used the average frequency of uses for substitution, augmenta- 
tion, student-teacher displaying activities, teacher-assigned activities, after-school remediating activi- 
ties, and learning aid activities as independent variables. We controlled for student physics interest 
as it might be highly correlated with students' achievement (Chang & Cheng, 2008). We also 
controlled for gender and pre-score. To account for classroom differences (students were in 28 
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classrooms), we included a classroom-level covariant—mean frequency of use for each class- 
room to control for the confounding variable that influenced both students’ use of technology 
and their achievement due to which classroom the student was in. We applied HLM with random 
intercepts (Raudenbush & Bryk, 2002) to analyze the data (statistic model refers to supplemen- 
tary material 2). For ease of interpreting the magnitude of the effects, we standardized all vari- 
ables into z-scores prior to analysis (M = 0, SD = 1). 


Results 

With controlling for the pre-score, we conducted an analysis of variance on achievement scores 
using the classroom as a fixed independent variable. Results indicate a significant main effect 
of classroom on student post achievement, y? = 67.82, df = 27, p < 0.001. Additionally, there is 
a moderate ICC of 0.05, which means that 5% of the variance in student achievement can be 
explained by which classroom the student was in. This result indicates that it is necessary to 
employ HLM analysis to control a Type I error. 


Overview of the effect sizes 

Our first research question aims to explore the relationship between the levels of mobile technol- 
ogy use and student achievement. To examine the correlation of frequency of mobile technology 
use with physics achievement, we first ran the multilevel random-intercept model by entering 
pre-score, gender, physics interest, and mean class use into the model. We found the residual 
variance for level 1 was oo? =. 457 and for level 2 was v? = .032. After entering the two vari- 
ables of the substitution uses and the augmentation uses into the prior model, we found the resid- 
ual variance changed to c4? =. 483 for level 1 and to 712 = .031 for level 2. We then calculated 
the explained proportion of the mobile technology uses of both substitution and augmentation by 
employing the definition of proportions of explained variance in two-level models (Snijders & 
Bosker, 1999). 


2 2 

2 var ( Yi = Èh Xp ) ‘a TI 
R“ =] ——— = l -——— (1) 

var ( Yj ) o +z 

0 0 


The results indicate that the mobile technology uses (eg, substitution and augmentation uses) in 
this model could explain 5% of the total variance. 


Further analysis found that the augmentation uses had a significant positive correlation with stu- 
dent achievement (y5o = .244, p < .01), which means that student achievement will increase .244 
standard deviation when the frequency of augmentation uses increases one standard deviation. 
However, we did not find a significant correlation of the substitution uses with student achieve- 
ment (749 =—.081, p > .05) (see Table 2). This result confirms our hypothesis—the level of mobile 
technology use predicts the physics learning outcome and a higher level of mobile technology use 
yields a greater achievement. 


In addition to this major finding, we also found several other results. First, mean use for each 
classroom only accounts for 396 of the second-level variance and does not have a significant effect 
(Yo1 = —.006, p > .05). However, we found that student pre-score could positively predict student 
achievement (719 = .706, p < .01). Second, we found that student interest could positively predict 
their physics achievement while controlling for the other covariates (y3o = .187, p < .01). Finally, 
student gender does not predict student physics achievement (y5o = —007, p > .05). 
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Table 2: The overall effects of mobile technology with different levels of uses 
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Fixed effect Coefficient S.E. T-ratio df p 
For INTRCPTI 

INTRCPT2, yoo 0.015 0.040 0.391 26 .699 

CLASSUSE, yo —0.006 0.034 —0.165 26 .871 
For PRESCORE slope 

INTRCPT2, 719 0.706 0.030 23.573 791 .000 

CLASSUSE, y; -0.106 0.031 4999 791 001 
For SEX slope 

INTRCPT2, 729 —0.007 0.031 —0.226 791 822 

CLASSUSE, y; —0.016 0.020 -0.819 791 413 
For INTEREST slope 

INTRCPT2, yao 0.187 0.020 9.235 791 .000 

CLASSUSE, ya 0.001 0.018 0.071 791 944 
For SUBSTITUTION slope 

INTRCPT2, yao —0.081 0.047 -1.738 791 .082 

CLASSUSE, ya —0.027 0.048 —0.563 791 573 
For AUGMENTATION slope 

INTRCPT2, 759 0.244 0.043 5.686 791 .000 

CLASSUSE, 751 —0.034 0.039 —0.878 791 .381 
Random Effect SD Variance df x p 

INTRCPTI, ese 0.180 0.032 26 77.541 .000 

level-1, a, 0.676 0.457 


The effect size for each type of augmentation uses 

In light of the positive effect size of augmentation uses, we further examined the relationship be- 
tween the frequency of using mobile technology in physics learning across the four different 
types and student physics achievement. We used the same HLM model in this analysis but re- 
placed the augmentation uses with its four types of use. We found the residual variance decreased 
to o2? = .438 for the first level and 71? = .029 for the second level. We then calculated the ex- 
plained proportion of the mobile technology uses of both substitution and the four types of aug- 
mentation uses by employing formula (1) and found that mobile technology accounted for 1096 of 
the total variance in this model. 


Further examination indicates that the results are similar to the prior analysis regarding the 
covariates (see Table 3). However, for our primary concern about the frequency of the substitution 
uses and the augmentation uses, there are several interesting findings: the substitution uses still do 
not have a significant effect size (y49 = —061, p > .05) as in the prior analysis, but the augmenta- 
tion uses display diverse effect sizes across different types of uses. 


First, the uses of student-teacher displaying activities (ys = .090, p < .05) and the after-school reme- 
diating activities (y7q = .214, p < .01) are positive predictors of student achievement, and the latter 
has a greater magnitude of the effect size than the former. Second, the uses for learning aid activ- 
ities also have a significant correlation with student achievement, but the coefficient is negative 
(yso * —086, p < .05). This finding means that higher use of mobile technology in these activities 
would be associated with lower physics achievement. Third, we found the interaction between 
mean use of each classroom and the uses for student-teacher displaying activities had a negative 
effect on student physics achievement (ys; = —094, p « .05). This finding means that higher 
use of mobile technology for student-teacher displaying activities in lower mean use classrooms is 
associated with higher achievement scores. In addition to these significant effects, we also found 
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Table 3: The discrete effects with different pedagogical uses for augmentation level 


Fixed effect Coefficient S.E. T-ratio df p 
For INTRCPT1 

INTRCPT2, yoo 0.016 0.040 0.414 26 682 
CLASSUSE, yoi —0.007 0.034 —0.217 26 .830 
For PRESCORE slope 

INTRCPT2, 710 0.735 0.034 21.753 785 000 
CLASSUSE, y11 —0.098 0.034 -2.911 785 .004 
For GENDER slope 

INTRCPT2, y20 —0.002 0.030 —0.081 785 .936 
CLASSUSE, y21 —0.019 0.020 —0.957 785 339 
For INTEREST slope 

INTRCPT2, 730 0.196 0.020 9.990 785 .000 
CLASSUSE, 731 —0.001 0.020 —0.073 785 942 
For SUBSTITUTION slope 

INTRCPT2, 740 —0.061 0.044 -1.381 785 .168 
CLASSUSE, 741 0.010 0.049 0.196 785 845 
For AUGMENTATION-S slope 

INTRCPT2, yso 0.090 0.039 2.282 785 023 
CLASSUSE, 751 —0.094 0.041 —2.300 785 .022 
For AUGMENTATION-T slope 

INTRCPT2, 760 0.030 0.025 1.202 785 .230 
CLASSUSE, 761 —0.003 0.026 —0.105 785 .917 
For AUGMENTATION-A slope 

INTRCPT2, 779 0.214 0.041 5.160 785 .000 
CLASSUSE, 71 —0.001 0.038 —0.035 785 972 
For AUGMENTATION-L slope 

INTRCPT2, yao —0.086 0.033 -2.591 785 .010 
CLASSUSE, ys1 0.011 0.034 0:325 785 .745 
Random Effect SD Variance df x p 
INTRCPTI, esc 0.171 0.029 26 71.441 .000 
level-1, a; 0.662 0.438 


no significant correlation between student physics achievement and the uses of teacher-assigned 
activities (yo = .030, p > .05). In all, our results suggest that although the augmentation uses have 
a higher effect size than the substitution uses, the effect is different across four types of uses. 


Discussion 

Despite a general assumption that a higher level of technological integration would result in 
greater student achievement (Puentedura, 2006), there is a lack of empirical evidence to sup- 
port this assumption with mobile technology. Our data empirically tested this assumption 
in high school physics learning. The substitution uses, which only substituted for the conven- 
tional instructional methods in physics classrooms, did not offer functional improvement and 
were found irrelevant to student physics achievement. However, the augmentation uses, which 
provided functional improvement compared to conventional instructional methods, were found 
to have a significant positive correlation with student physics achievement. This finding sug- 
gests that teachers and students might benefit from a higher level of mobile technology use in 
physics learning, consistent with the general assumption about technology-integrated learning 
(Geer, White, Zeegers, Wing, & Barnes, 2017; Romrell et al., 2014). 


© 2018 British Educational Research Association 


14 British Journal of Educational Technology Vol 0 No 0 2018 


Although on average, augmentation uses were found to be positively correlated with students' 
physics achievement, this correlation was not identical across different pedagogical approaches. 
This finding was consistent with the findings from the study conducted by Lindsay (2016), who 
found the one-to-one mobile technology had great potential in classroom learning but depended 
on the pedagogical approach. The result suggests that in addition to the levels of integration, ped- 
agogical factors also play an important role in the learning outcome (Harris, Mishra, & Koehler, 
2009). One of the four pedagogical approaches, the after-school remediating activities, had the 
greatest effect size on students' achievement, as compared to the other three patterns. There are 
two possible explanations for this finding. First, according to the constructivist learning theory 
that knowledge has to be constructed by learners themselves, instructions or learning tools that 
support the knowledge construction process should have a great impact on student learning 
(Lan, Tsai, Yang, & Hung, 2012). The after-school remediating activities involves four specific uses 
that aim to facilitate the knowledge construction process. The building incorrect item set and using 
incorrect item set function allow students to collect and review problems that they have difficulty 
solving. (After-school) Class note helps students review the content learned in class, and Mindmap 
helps students create a mental structure for the content. Second, all four uses in this category 
were initiated by learners themselves and happened during self-directed learning. They are all 
after-school uses and not depending on teachers' instruction about when and how they should 
use them. This finding means that whether or how to use them is dependent on the students, 
which features a self-regulated learning process. According to Pintrich (2004), the higher-perfor- 
mance students in self-regulated learning usually possess a higher self-efficacy or learning moti- 
vation; thus, might use these functions more frequently. For example, the students with higher 
self-efficacy in learning might use the Mindmap function more frequently, whereas the students 
with lower self-efficacy might use it infrequently, if at all (Merchie & Van Keer, 2012). The self- 
regulated learning that grants students more freedom to use the mobile technology according to 
their needs might be another reason for the high effect size. 


Student-teacher displaying activities also facilitated student physics achievement, although the 
effect size is smaller in magnitude compared to after-school remediating activities. These activities 
involve student-teacher interactions, with the mobile devices helping teachers collect informa- 
tion about the students' learning process. For example, Clicker and Class test provide teachers with 
statistical results that reflect whether or to what extent students understand physics concepts. 
With Doodling and Screen display, teachers are able to review student work and then to select 
some of them to critique or share with other students. These tools help teachers better under- 
stand their students' learning process, allowing them to adjust instruction to accommodate stu- 
dent needs. According to Harris, Mishra, and Koehler (2009), this advance in teaching probably 
due to an increase in teachers' technological pedagogical content knowledge (TPACK). We also 
speculate that students in these activities were more active because, in order to respond to the 
teacher's request, students had to think carefully or apply the content they had just learned. 
These processes may involve more students in collaboration with teachers and engage them in 
active knowledge construction, thus facilitating their learning. 


In addition to these positive correlations of technology use with student achievement, we also 
found several negative correlations that are relevant to student achievement. First, the interac- 
tion between the average use of each classroom and the uses for student-teacher displaying activ- 
ities has a negative correlation with student achievement. Considering that the student-teacher 
displaying activities are found to have a positive correlation with student achievement, this finding 
could be explained by the fact that the student-teacher displaying activities tends to be effective in 
the lower mean use classrooms, whereas in the classrooms with higher mean use, the correla- 
tion tends to be minimum. It suggests that a higher frequency of uses does not always indicate a 
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greater achievement. Our finding that students who used it most frequently had lower achieve- 
ment might also be explained as that higher use of mobile technology is potentially distracting 
to student learning. This point is consistent with the findings from Tossell and his colleagues 
(2015), who found some students perceived mobile devices as a distractor in classroom learning 
if overused. This assertion is also supported by other evidence regarding learning aid activities. We 
found the frequency of learning aid activities is negatively correlated with student physics achieve- 
ment. A close look at the three specific uses reveals that they are used mostly for learning material 
accumulation during other activities rather than knowledge construction. It seems that frequent 
use of these functions might distract students from learning. For example, frequent use of building 
or using incorrect item set in the classroom might distract students and consequently influence 
students’ other learning activities. In addition, misuse might also account for these negative cor- 
relations. For example, the Homework guide—one of the learning aid activities, might be effective if 
students used it after having completed their homework. However, if students used the resource 
before completing the homework, the Homework guide became a plagiarizing tool. The students 
would simply borrow the ideas or answers from the guide without spending time thinking about 
and practicing on their own. Therefore, this use may thwart student learning. 


Finally, we found that teacher-assigned activities did not significantly correlate with student 
achievement. In some of these activities, mobile technology might benefit teachers but not stu- 
dents. For example, My homework, which collects student data immediately, saves time for teach- 
ers, but might not yield many benefits for students compared to traditional assignments. In other 
activities, students might be dutifully involved in the activities, but not actively constructing 
knowledge. Using learning guide is such an example; through this guide, students could watch text 
materials, pictures, or videos, but only when teachers require them to do so. As a result, these 
uses may not contribute much to learning. 


Conclusion and limitations 

By examining 803 high school students’ uses of 20 specific functions in physics learning for half 
a year, we found that the uses considered as a higher level of mobile technological integration 
(augmentation) had a significant positive correlation with student achievement, whereas the uses 
with a lower level of integration had no significant correlation. Further analysis showed that the 
pedagogical approach mediated the effect of the technological integration even within the same 
level. After-school remediating activities and student-teacher displaying activities that support self- 
directed knowledge construction and collaborative learning are positively correlated with student 
physics achievement, in contrast to teacher-assigned activities with no correlation and learning aid 
activities with a negative correlation. 


We also noted some limitations in the present study. First, even though we employed HLM in the 
study to control some confounding variables that due to students sitting in different classrooms, 
we did not randomly assign students in different groups and controlled the independent variables. 
Future studies could employ randomized controlled experiments to further verify the assumption. 
Second, though examining 20 specific mobile uses in physics learning, we only categorized the 
uses into the two lower levels in the SAMR model. Although this might be due to the current 
status of mobile technology use in physics learning, considering this model has four levels, future 
research should examine mobile technology uses that manifest higher SAMR levels to help us 
better understand the relationship between the level of mobile technology use and physics learn- 
ing. Third, our assumption—a higher level of mobile technology use predicts a greater effect— 
was drawn from broad educational technology literature, but the current study only verified it 
in the domain of high school physics learning. Future research should collect data from other 
subject areas to further verify this assumption. Last, concerning the limitation of quantitative 
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studies, although we provided some explanations regarding the results of the four patterns of 
augmentation uses and their effect, we did not collect direct evidence from students and teachers 
regarding their motivation for use to support these explanations. Future studies should employ 
qualitative or mixed methods to explore the reasons underlying these results. 
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The original dataset is available upon request. 

The dataset does not include students' personal information, and the data collection received 
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